Code generation from UML / MARTE / OCL Environment Models to Support Automated System Testing of Real - Time Embedded Software

Given the challenges of testing at the system level, only a fully automated approach can really scale up to industrial real-time embedded systems (RTES). Our goal is to provide a practical approach to the model-based testing of RTES by allowing system testers, who are often not familiar with the system’s design but are application domain experts, to model the system environment in such a way as to enable its black-box test automation. Environment models can support the automation of three tasks: the code generation of an environment simulator to enable testing on the development platform or without involving actual hardware, the selection of test cases, and the evaluation of their expected results (oracles). From a practical standpoint―and such considerations are crucial for industrial adoption―environment modeling should be based on modeling standards (1) that are at an adequate level of abstraction, (2) that software engineers are familiar with, and (3) that are well supported by commercial or open source tools. In this paper, we propose a precise environment modeling methodology fitting these requirements and discuss how these models can be used to generate environment simulators. The environment models are expressed using UML/MARTE and OCL, which are international standards for real-time systems and constraint modeling. The presented techniques are evaluated on a set of three artificial problems and on two industrial RTES. Key Words and Phrases: Environment modeling, environment simulation, automated testing, modelbased testing, real-time embedded systems, search based software engineering 1. Introduction Real-time embedded systems (RTES) are widely used in many different domains, as for example from integrated control systems to consumer electronics. Already 98% of computing devices are embedded in nature and it is estimated that, by the year 2020, there will be over 40 billion embedded 1 This paper is an extension of a conference paper “Environment Modeling with UML/MARTE to Support Black-Box System Testing for Real-Time Embedded Systems: Methodology and Industrial Case Studies” published in proceedings of ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS), 2010 [1] Simula Research Laboratory Technical Report (2011-04) Version 3.0 2 computing devices worldwide [2]. Testing these systems such that they are functionally correct and do not lead their environment into critical states (e.g., unsafe) is vital. RTES environments typically comprise a number of physical components (e.g., sensors and actuators) and possibly other RTES systems (e.g., in systems of systems). Typically, there is a large number and variety of stimuli to the RTES with different patterns of arrival times. These characteristics make the testing of RTES challenging and increase the need for automated, systematic testing strategies. Because RTES are developed for diverse domains presenting different constraints (e.g., different timing, safety, security requirements), different testing approaches are required to handle the varying set of characteristics required by these domains [3]. Our main target RTES in this paper are soft-real time systems with time deadlines in the order of hundreds of milliseconds with an acceptable jitter of a few milliseconds in response time. Our testing approach (black-box system level testing) not only encompasses functional correctness of the system under test (SUT), but also enable to focus testing on particularly critical aspects of the RTES, i.e., potentially hazardous situations. Typically, large scale testing of RTES software in real environments and on actual deployment platforms is not a viable option. It would be expensive, the consequences of failures might be catastrophic (e.g., in safety critical systems), and the number of variations in the environment that can be exercised within a reasonable time frame are small. Moreover, some of the environment components might not be available at the time of testing, since hardware and software components are typically developed concurrently. To test RTES software in this kind of situations, a common strategy is to develop a simulator for these environment components. When testing RTES, the simulation of three concepts (or their combinations) is typically considered: the SUT, its hardware platform, and the environment with which the SUT interacts. Depending on the goal of testing, different combination of these three concepts can be simulated [3]: (i) at early stages of the development process, a typical approach is to model and simulate the SUT, its hardware and its environment to ensure that the specifications of the SUT do not violate the environment assumptions; (ii) the embedded software is tested on the development platform with a simulated environment to ensure that the developed software works according to the environment assumptions and can handle possible environment failures. This is done with either an adapter for the hardware platform that forwards the signals from the SUT to the simulated environment or a simulation of the hardware platform; (iii) another level of simulation is when the actual software is deployed on the hardware platform (or part of the platform, e.g., only the processor) and testing is done with a simulated environment. The focus of this paper is on the second type (ii) of modeling and simulation in which the actual SUT is used, the environment is simulated, the hardware platform is simulated or bypassed through an adapter communicating with the environment simulator. In our experience of working with two industrial organizations, which were developing RTES for different domains (seismic acquisition Simula Research Laboratory Technical Report (2011-04) Version 3.0 3 systems and automated bottle recycling machines), this form of testing was highly critical as it enabled early verification of the RTES. To address the above objective, in this paper, we propose an automated methodology for RTES based on environment behavioral models developed using software modeling standards: Unified Modeling Language (UML) [4], UML Profile for Modeling and Analysis of Real-time and Embedded Systems (MARTE) [5], and Object Constraint Language (OCL) [6]. The main contributions of this paper include an environment modeling methodology and an approach to generate a simulator of the environment from the environment model in a way to enable the automated testing of industrial RTES. As further discussed below, our focus is to devise a practical approach in a system testing context, and we evaluate both the modeling methodology and simulation generation on two industrial case studies. Environment models describe both the structural and behavioral properties of the environment. Given an appropriate level of detail, defined by our methodology, they enable the automatic generation of the environment simulator. These models can also be used to generate automated test oracles, which are typically modeled as “error states” that should never be reached by the environment during the execution of a test case. Moreover, the models can further be used to automatically select test cases and sophisticated heuristics are used to automatically do so from the models without any intervention of the tester. To summarize, the only required artifacts to be developed by testers is the environment model and the rest of the process is expected to be fully automated. Incidentally, by using this automated Model-Based Testing (MBT) technology, one of our industrial partners was able to find new critical faults in their RTES. To support environment modeling in a practical fashion, we have selected standard and widely accepted notation for modeling software systems, the UML and its standard extensions. We use the MARTE [5] extensions for modeling real-time features and OCL for specifying constraints. We have also provided lightweight extensions to UML to ease its use in our context. As we will discuss later, environment modeling is not a new concept. But, most of the approaches use non-standardized notations or grammars for modeling, which makes them difficult to apply from a practical standpoint. Modeling the environment of industrial RTES systems using a combination of UML, MARTE, and OCL has not been addressed in the literature. By using the proposed methodology, the software testers (who are primarily software engineers) can model the environment with a notation that they are familiar with, using commercial or open source tools, and at a level of precision required to support automated MBT. The importance of relying on standards for modeling was confirmed on the two industrial case studies across entirely different domains. Although code generation from models has been widely studied, the context of black-box RTES system testing poses specific challenges and problems that are not fully discussed and addressed in the literature. For this purpose we present extensions to the state pattern [7] specifically aimed at Simula Research Laboratory Technical Report (2011-04) Version 3.0 4 enabling environment simulation for system testing and define rules for transforming environment models to Java code (the simulator). To summarize, the fundamental motivation here is that system testers, in many industry sectors, are usually application domain experts but have little or no knowledge of the system design and implementation. Our approach is therefore black-box and does not require the RTES itself to be modeled. It only requires its environment to be modeled at the right level of abstraction and in such a way as to enable effective test automation. The reliance on software modeling standards offers significant advantages, such as the possibility of using (1) different commercial and open source modeling tools (e.g., IBM Rational Software Architect (RSA), Papyrus, or Enterprise Architect), (2) notations that many software engineers—including system testers—might already be familiar with and that can be used to also model the SUT, and (3) existing analysis tools (e.g., [8]) that can take such models as input. The paper is organized as follows: Section 2 shed light on the practical motivations and aspects of the work presented in this paper, setting the context to better justify our approach; Section 3 discusses the related work. Section 4 presents the motivating example that we use throughout the paper to explain various concepts. Section 5 discusses the proposed environment modeling methodology. Section 6 goes into the details of the most important decisions regarding the transformation of models to simulation code, whereas Section 7 presents the case studies. Section 8 discusses the limitation of the proposed work and finally, Section 9 concludes the paper. 2. Practical Aspects The work discussed in this paper was motivated by the problems faced and practices followed by two industrial organizations that we worked with, namely WesternGeco AS, Norway and Tomra AS, Norway. These two organizations were developing RTES for two different domains; WesternGeco was developing a seismic acquisition system and Tomra was developing automated recycle machines. Both the RTES were developed to run in an environment that enforces time deadlines in the order of hundreds of milliseconds with an acceptable jitter of a few milliseconds in response time. In one of the organizations, testing the SUT on the development platform with a simulated environment was considered to be mandatory before deploying the software on the operational hardware. To achieve this, software engineers were writing application specific simulators directly in Java. Test cases for system level testing were written by hand by the software test engineers and were executed on the SUT with the environment simulator. The research presented in this paper was strongly driven by our investigation of the practical needs of our industry partners which, based on our experience, are shared by many others in numerous industry sectors. Our understanding of these needs is presented in the remainder of this section. 2 Webpage: http://www.ibm.com/developerworks/rational/products/rsa/, date last accessed: 05/02/2012 3 Webpage: http://www.papyrusuml.org/, date last accessed: 05/02/2012 4 Webpage: http://www.sparxsystems.com.au/, date last accessed: 05/02/2012 Simula Research Laboratory Technical Report (2011-04) Version 3.0 5 Manually writing an environment simulator using a programming language (e.g., Java or C) appeared to pose a number of issues, the main one being that software engineers have to develop such simulator at a low-level of abstraction while simultaneously focusing on the logic of the simulator, complex programming constructs (e.g., multiple threads, handling timers), and the handling of test case configurations (when the simulator is used for testing). Making this problem even more acute, over the course of the RTES development, these simulators frequently change due to changes in the specifications of the hardware components. Typically, modeling and simulation (M&S;) approaches focus on simulating hardware components, execution platforms, and natural phenomena in the RTES environment using various simulation tools, such as DEVS [9] and Modelica [10]. These M&S; tools support precise simulations of both discrete and continuous system behaviors and are typically based on mathematical models. However, in our context, such M&S; tools are not practical, for a number of reasons: (i) Software engineers, who are typically in charge of system testing at this level, are often not familiar with such simulation languages. To enable technology transfer in industrial practice, it would be more convenient for them to develop or generate the simulator using a language that they are familiar with, as for example the languages used to program and model the SUT; (ii) These simulation tools do not support automated environment-based testing of RTES software. A number of features must be modeled to enable this kind of testing. For example, the models need to provide information for the automated generation of oracles (to verify whether test cases pass or fail). Furthermore, the simulator needs to interact with a test harness to get appropriate values for various non-deterministic events. The exact occurrences of such events in the environment cannot be determined. These events may follow different probability distributions (e.g., probability of failure of a sensor) or may occur at any time within a given time interval (e.g., a gate at a railroad intersection may take from 5 – 7 seconds to close); (iii) Another issue is that in simulation languages such as ModelicaML [11] and DEVS [12], since they were developed for a different purpose, there are limitations regarding the interactions of the simulator with the production code of the RTES (e.g., handling of operating system resources, such as inter-process communication with the production code over TCP/UDP). Such an interaction is a requirement for the type of testing we deal with in this paper, since the environment simulator has to interact with the actual RTES production code to receive stimuli and to send responses. In dealing with such interactions, we do not want any constraint regarding the programming language in which the RTES is written. The modeling methodology presented here provides an automated model-based approach to derive environment simulators, test cases, and test oracle, taking into consideration all the practical aspects described above, which are common place in many industrial environments. The only major input required are the environment models describing the structure and behavior of the environment. Since the intended users are software engineers, we chose standard software modeling languages for environment modeling with the aim to make the modeling methodology as simple as possible. This Simula Research Laboratory Technical Report (2011-04) Version 3.0 6 paper discusses the methodology for modeling the environment based on the selected modeling standards and a specific profile and furthermore describes the process of simulation generation from those environment models. It does not, however, address in detail test generation and test oracles. 3. Related Work In this section we discuss the related literature in the areas of (1) modeling and simulation for RTES Testing, (2) environment modeling and environment model-based testing of RTES, and finally (3) code generation approaches from UML state machines and class diagrams. 3.1. Modeling & Simulation for RTES Testing As discussed earlier, based on the goals of testing of RTES, the SUT, the hardware platform, the environment, or their combinations can be modeled and simulated. At the early stages of the development process for RTES, a typical approach is to model and simulate the SUT, its hardware and its environment. The aim is to ensure that the model of the SUT complies with the requirement specifications and does not violate the environment and hardware assumptions. This approach is sometimes also referred as “model-in-the-loop” simulation [3, 13, 14]. Another level of simulation for testing is when the actual executable software is deployed on the real hardware platform (e.g., electronic control unit) and their combination is tested with a simulated environment (e.g., with the simulation of plant model [3]). This approach is generally referred to as hardware-in-the-loop testing [15, 16]. Typically, a prototype of the hardware platform is used at this stage. A variation to hardware-in-the-loop testing is the case where only the actual processor is used during testing and rest of the hardware and environment are simulated. This variation is referred to as processor-in-the-loop testing [17]. Before the hardware or the processor is available, the embedded software can also be tested on the development platform (e.g., Linux or Windows-based machine) with a simulated environment and hardware platform. This is typically done to ensure that the developed software works according to the environment assumptions and behave appropriately in hazardous or abnormal situations. This is mostly referred to as software-in-the-loop simulation [3, 13]. Existing modeling and simulation languages have been developed and are widely used for the first three types of simulations. In these cases the environment simulation needs to interact with the actual hardware or its simulation. In such cases, precise simulation of both discrete and continuous phenomena is required and is typically based on mathematical models. In this paper, we target a slight variation to the typical software-in-the-loop simulation. We only model and simulate the environment and use an adapter for the hardware platform that forwards the signals from the SUT to the simulated environment. Our research problem definition is motivated primarily by the practical needs of our industrial partners (Section 2) but it is expected to be relevant in many other industrial environments developing similar RTES. Other approaches that do testing with a similar focus are discussed next. Simula Research Laboratory Technical Report (2011-04) Version 3.0 7 3.2. Environment Modeling and Environment Model-based Testing In this subsection, we will discuss various environment modeling and model-based testing approaches discussed in the literature. Kishi and Noda [18] present an approach for modeling the environment of an embedded system using an aspect-oriented modeling technique. Karsai et al. [19] propose a new language for modeling the environment of an embedded system. Choi et al. [20] use annotated UML class and sequence diagrams for modeling and simulation of environment. Kreiner et al. [21] present a process to develop environment models for simulation of automatic logistic systems and its environment. Axelsson [22] evaluates how UML can be used to model real-time features and provides extension to UML for modeling of real-time systems and their environments. Gomaa [23] discusses the use of a context diagram for modeling the relationship between an RTES and its external entities. Friedentahl et al. use the concept of SysML block diagram and activity diagrams to represent the system and its interfaces with environment components [24]. There are a few works reported in literature that discuss testing of RTES based on its environment and without considering the hardware platform or its simulation. Auguston et al. [25] discuss the development of environment behavioral models using Attributed Event Grammar for testing of RTES. The behavioral models contain details about the interactions with the SUT and possible hazardous situations in the environment. These models are then traversed to obtain various test scenarios. The approach is applied on a simulation of the RTES specifications. Heisel et al. [26] propose the use of a requirement model and an environment model using UML state machines along with the model of the SUT for testing. Adjir et al. [27] discuss a technique for testing RTES based on a model of the system and intended assumptions in the environment in Labeled Prioritized Timed Petri Nets. Larsen et al. [28] propose an approach for online testing of RTES based on time automata to model the SUT and environmental constraints. Bousquet et al. [29] present an approach for testing of synchronous reactive software by representing the environmental constraints using temporal logic. To summarize, there are approaches in literature that deal with modeling the environment of a system for various purposes. Most of these approaches are only limited to modeling the static structure of the environment, as they do not focus on test automation. The approaches that deal with modeling of behavioral aspects either use notations with which software engineers are not familiar, or provide extensions for environment modeling that in a non-standard way. All environment modeling approaches aimed at supporting testing in literature, except the one by Heisel et al. [26], use non-standard languages for modeling. This work, however, models the concepts of probabilities and time using non-standard notations, without relying on UML extension mechanisms. Furthermore, most of the works on environment model-based testing, model both the SUT and the environment, which does not fit our purpose: black-box, system testing. Moreover, none of these works simulate the behavior of the environment. To be able to execute different possible behaviors of the environment based on its interaction with SUT for system testing, a simulator is essential. Generating a random set of scenarios from the environment models, as done by Auguston et Simula Research Laboratory Technical Report (2011-04) Version 3.0 8 al., [25] provides a limited coverage of the environment model. Last but not least, none of the relevant works assess their environmental methodology on an actual RTES system, which we believe is a requirement to assess the credibility and applicability of any MBT approach. 3.3. Code Generation from UML Classes and State Machines There is no reported approach in the literature that generates RTES environment simulators from UML models or their extensions. As we discussed in Section 3.2, though modeling and simulation languages and environments have been proposed, they do not fit our purpose of black-box system testing (Section 2). Generating code from state machines is not a new problem. Even though a number of works are reported in the literature (e.g., [30], [31]) and a number of tools are available that generate code from state machines (e.g., SmartState [32], IBM Rhapsody [33]), no technology was developed having the required features for black-box testing based on environment models, such as non-determinism (a common feature of the environment) and test-specific behavior (e.g., modeling illegal or unsafe environment states). The use of standards is also an important requirement for our modeling methodology, as discussed earlier. The modeling standards that we selected for our methodology are supported by a wide range of tools and support is available for training. The existing code generation tools and techniques discussed in the literature are focused on generating system code and not environment simulators. They are not applicable to our purpose because of several reasons. Following, we discuss the most important ones: 1) The models need to capture specific states of failures in the environment components (the “failure states”, e.g., a sensor break down). The environment simulators then must be able to simulate these situations, which occurrences can be controlled by the test cases. Similarly, the models need to capture information of situations in the environment that should never happen if the SUT is correct (which we call the “error states”). Such information is required to derive test oracles and to guide testing strategies. 2) Since we simulate the environment for testing, the generated code is strongly coupled with the test harness, which is not the case with existing simulators. 3) The generated simulator includes code that collects execution information to guide testing strategies. Such information is used in heuristics to generate test cases that are effective to reveal faults. The heuristics that we use are specific to our modeling and testing methodology, existing tools for code generation do not provide this support. 4) Extensions to existing tools are not feasible in most cases as some of them are proprietary and others are not based on model-driven engineering standards, such as the Eclipse Modeling Framework. The original state pattern, discussed in [7], provided a design pattern to implement state-driven behavior but did not address a number of important features present in UML 2.x state machines, e.g., concurrency, time events, change events, and actions. A number of extensions for the pattern have Simula Research Laboratory Technical Report (2011-04) Version 3.0 9 been discussed over time to handle missing features (e.g., [34, 35]). Most of these extensions are focused on increasing the understanding and usability of the code obtained using the state pattern to support programmers (e.g., [35]) and are not very useful in our context where the code is automatically generated from the models. Chin and Millstein [36] propose an extension to the state pattern for handling state behavior in inherited sub-classes. Holt et al. [37] propose an extension for handling state and transition actions and report its application on an industrial case. Palfinger [38] provides an extension to the state pattern that allows extension of object’s behavior at runtime by using a mapper class. In our approach, this was not applicable as we do not require the addition of new behavior during the execution of environment components. The work in [39] extends the state pattern to supports hierarchical state machine and time events. We handle time in a similar way, i.e., by using a separate timer class that calls timeout()on the context object in case of a timeout. The approach in [39] does not handle parallel regions and change events, does not provide details on handling actions, and does not provide support for the test-related features required by our approach. Overall, none of the extensions of the state pattern in the literature completely meets the needs for RTES environment simulation to support system testing. We could have used some optimizations to improve ease of understanding and modification, and cleaner code generation, but these would not have had a large impact as the generated source code is not visible to the end-user and is only provided as an executable archive. Furthermore, there are optimizations in the literature to improve the performance of the generated code (e.g., minimizing the number of running threads to avoid overheads due to context switching). However, in our framework (as we have explained in details throughout the paper) we do not need to optimize performance. The generated simulators are used only for testing purposes, and each test case runs on a different process, lasting from a few seconds to a few minutes (depending on the RTES). As long as the environment simulators can behave as expected (e.g., providing the right stimuli at the right time), this would be sufficient for our testing purposes. This was the case for all our case studies. 3.4. Summary To summarize, this paper differs from existing works in several of the following ways: (1) It provides an environment modeling methodology based on international software engineering modeling standards (UML 2.x, MARTE, OCL) that is dedicated to black-box, RTES system testing. The targeted RTES have complex environments and have soft-real time constraints in the order of hundreds of milliseconds pertaining to the response time of the SUT and operations of the environment. This is the first work focusing on such methodology, allowing the modeling of important concepts for testing such as modeling non-determinism and oracle information, while relying only on light-weight, standard extensions of UML (i.e., by defining a UML profile); (2) It provides an approach to generate simulators, based on environment models developed using the proposed environment modeling methodology, addressing the specific needs of black-box system Simula Research Laboratory Technical Report (2011-04) Version 3.0 10 testing; (3) Regarding simulator generation, the paper provides extensions to the state pattern that handle time-related features and various UML 2.x features that were not previously discussed in the literature, including handling of change events and concurrency; (4) Unlike most of the works reported in the literature, this paper assesses the proposed methodology and simulator generation on two industrial RTES, which we believe is a requirement to assess the applicability of any test automation approach. 4. Motivating Example To motivate and explain our modeling methodology and simulator generation strategy for RTES, we take as example a subset of one of our industrial case studies. Note that we have sanitized the information due to confidentiality restrictions. This example is an Automated Bottle Recycling System developed by Tomra AS, Norway. It is representative of the type of RTES we are targeting in this paper: it has soft-real time constraints in the order of hundreds of milliseconds pertaining to the response time of the SUT and operations of the environment (e.g., the sorting of item should be done within a couple of minutes after an item is inserted). The portion of the case study considered in this paper is focused on the important functionality of sorting the recycled items to their proper storage locations (or destinations). Users insert the items to be recycled inside the front-end of the recycling system, called the Reverse Vending Machine (RVM). The items can be of three different types for the subset we are discussing: plastic bottles, cans, or glass bottles. The RVM forwards the items to the Sorter, which is a sorting arm (we only consider a simplified backroom with a single sorting arm). On its way from an RVM to Sorter, an item can be lost if it is not detected in time or if it falls from the moving belt. The Sorter can move in three directions (each leading to a specific destination) and its movement is controlled by a Sorting Controller. The Sorting Controller is the system under test in our case study. The Sorting Controller receives information of the type of the item inserted from the RVM and when it is supposed to reach the Sorter. The Sorting Controller is responsible for moving the Sorter in a position that leads the items to their appropriate destinations. There can be different destinations based on the type of items. Plastic bottles and cans are placed in their appropriate bins, whereas the glass bottles are placed in the crates. The Sorting Controller should prevent certain erroneous situations from happening. For the subset of the case study discussed in this paper, we consider two such situations: (i) when an item is not correctly sorted and it goes to a wrong destination (for example, a plastic bottle going into a bin of cans) (ii) when an item reaches the Sorter while it is still moving. 5. Environment Modeling Methodology If environment models are to be used for testing RTES, they should not only be sufficiently detailed, but should also be easy to understand and modify as the environment and RTES evolve. To handle the complexity of realistic RTES environments, the modeling language should have provision Simula Research Laboratory Technical Report (2011-04) Version 3.0 11 for modeling at various levels of abstraction. The modeling language should also be well-defined for the tools to analyze the models and for the humans to accurately understand them. The language should also provide features (or allow possible extensions) for modeling real world concepts, realtime features, and other concepts, such as non-determinism, required by the environment components. The UML, MARTE profile, and the OCL together fulfill the important requirements of an environment modeling language. Even though we are using the same notations to model the environment that are used for modeling software systems, it is important to note that the methodology for environment modeling is significantly different from system modeling. While modeling our industrial cases, we abstracted the functional details of the environment components to such an extent that only the details visible to the SUT were included. An environment of a RTES typically features a number of non-deterministic events (e.g., breakdown of a sensor), which must be modeled. Such events are not common when modeling the internal behavior of a system. To model RTES environments, we have developed a profile that provides support for modeling various concepts central to our methodology and highlights the subset of UML/MARTE that is required for such modeling. For testing the system based on its environment, the behavioral details of the environment are as important as its structural details. Structural details of the RTES environment are important to understand the overall composition of the environment (e.g., number and configuration of sensors/actuators), the characteristics of various components, and their relationships. We choose to model these details in the form of a Domain Model developed using UML class diagrams annotated with our defined profile. The behavioral details of environment components are required to specify the dynamic aspects of the environment, for example, to determine the possible environment states, before and after its interactions with the SUT, and to specify the possible Fig. 0. RTES Environment Modeling Profile Simula Research Laboratory Technical Report (2011-04) Version 3.0 12 interactions between the SUT and its environment. For behavioral details, we used the UML State Machines augmented with the MARTE profile and our defined profile. In the kind of testing this paper addresses, the focus is on the interactions of the RTES with the components in its environment, i.e., what are the possible inputs/outputs to/from the RTES from/to these components at any given point in time? How does the RTES behave in abnormal situations, such as a hardware failure in any of the environment components? A test case for a RTES would typically consist of a sequence of actions from the user(s), signals from/to sensors/actuators, and possibly hardware component breakdowns. This would correspond, in our context, to nondeterministic events that can happen during the environment simulations. In the following sub-sections, we discuss the environment modeling profile that we developed followed by the guidelines for modeling domain and behavioral models that were developed based on our experience of modeling two large-scale industrial RTES a marine seismic acquisition system and an automated bottle recycling system. 5.1. Environment Modeling Profile Our goal was to model the environment based, to the extent possible, on the standard UML and its existing extensions. We applied the standard notations and based on our needs for those case studies, where required, we provided light weight extensions to UML. In this section we will discuss the subsets of UML and MARTE that we used and the lightweight extensions that we have provided for environment modeling. From a practical standpoint, it was important to identify these subsets for the methodology, since the UML and MARTE standards are very large and most organizations would be reluctant to adopt such large notations. Developing UML profiles is a way to provide lightweight extensions to UML that do not conflict with its original semantics. To model an RTES environment, generate its simulator, test cases, and obtain test oracle from these models, we need more specific notations than what the standard UML provides. We provided extensions to the standard UML class diagram and state machine notations in the form of a profile. The profile also resolved various semantic variation points left open by the standard (discussed later in Section 6.5) to address our specific needs. Fig. 1 depicts a profile diagram for our proposed RTES environment modeling profile. The profile defines a set of stereotypes for modeling our methodology specific features on UML classes and state machines. It also shows the subset of MARTE that the profile is using, i.e., the Time package and the concept of GaStep from the Generic Quantitative Analysis Modeling (GQAM) package. The Time package allows the software engineers to model various time related features, such as timed events and action durations [5]. This small subset of UML and MARTE was sufficient for modeling our two industrial case studies for the purpose of automated black-box testing. 5.2. Domain Modeling Simula Research Laboratory Technical Report (2011-04) Version 3.0 13 Our environment modeling methodology for system testing requires the modeler to create an environment domain model that captures relevant structural details of the environment including the various components of the environment, their cardinalities, characteristics, and relationships. The domain model is developed using the UML class diagram notation. The various components modeled in the domain model together form the overall environment of the SUT. This means that all these components (their instances) will run in parallel with each other. The domain model represents various possible forms that the environment of RTES can take. Each component in the domain model can have a number of instances in the RTES environment. The information about the number of possible instances of a component in the environment is modeled as cardinalities on the associations between different components in the domain model. Therefore, the domain model can be used to obtain a number of potential configurations of the environment. To restrict the possible forms an environment of an RTES can take, OCL constraints can be specified. These constraints can for example be used to restrict the possible combinations of environment components or to restrict the possible values of attributes. The domain model for the Sorting Machine case study is shown in Fig. 2. Sorting Controller in the domain model is the SUT and the components RVM, User, Sorter and Item are the environment components. All the environment components are considered to be active objects, i.e., having their own thread of execution, and communicate with each other through signals. Each environment component in the domain model can have multiple instances. For example, in the domain model, shown in Fig. 2, Item is represented as one environment component, but during simulation it can have multiple instances. The number of instances to be created, which we refer to as an environment configuration, is determined based on the cardinality of relationships, i.e., in this case the cardinality of the association between User and Item with the role name itemCollection and the OCL constraints restricting the possible combination of environment components. In the motivating example we have restricted the possible number of items a user can enter to be less than 100. This is shown as an OCL constraint in Fig. 3. A valid environment configuration for this example is a single RVM, a single Fig. 0. Domain Model of Sorting Machine Case study Simula Research Laboratory Technical Report (2011-04) Version 3.0 14 Sorter and a User with three Items. A test case in our context is a combination of a setting of the simulator for the non-determinism in the environment models (e.g., a specific time at which a sensor stops working), which we call simulation configuration and an environment configuration. A test case uses these settings for a particular simulation run. During testing, the selected test strategy decides the way these configurations for a test case are generated by the Test Framework (e.g., random values for simulation and environment configurations when using random testing). Note that the domain model that we develop is different from the ones commonly discussed in literature (e.g., [40]). The components represented as classes in the environment domain model will not necessarily relate to software classes. They may correspond to systems, users and concepts related to various natural phenomena. Domain modeling here is not a starting point for software analysis. The identification of components in the domain model, their properties, and their relationships is also different from what is commonly done for software analysis. Following, we further discuss various guidelines for modeling the structural details of a RTES environment.
 5.2.1. Environment Components to be Included Initially, all the environment components that are directly interacting with the SUT are included in the domain model. Then, each of these components is further refined to a level where we are certain to cover the important details for simulating the environment needed to test the SUT. If at any time the behavior of an environment component is getting too complex, when possible, we can decompose the component and divide its behavior into multiple concurrent state machines. This is especially useful if a component can be divided into components that are similar to existing components, so that we can specialize existing state machines. The environment components in the domain are stereotyped with «Context». The environment components are modeled as active objects and can communicate with each other and the SUT through signals. 5.2.2. Relationships to be Included All those associations representing the physical or logical relationships among various environment components, or that were needed for components to communicate, should be included. A number of components in the environment might be similar to each other (e.g., various types of sensors). It is useful to relate these components (and their behavior) using the generalization/specialization relationship for simplifying the model, as our experience shows that such domain models get highly complex. For example, in the sorting machine case study, we modeled the association of the SortingController with the Sorter, which is controlled by the board. 5.2.3. Properties to be Included From all properties that may characterize environment components, it is important to include only context User inv: self.itemCollection→size() > 0 and self.itemCollection→size() < 100 Fig. 0. An example OCL constraint Simula Research Laboratory Technical Report (2011-04) Version 3.0 15 those properties that are visible to the SUT (or have an impact on a component that is visible to the SUT). These may include attributes that have a relationship to the inputs of the SUT, that constrain the behavior of a component with respect to the SUT, or that contribute to the state invariant of a component that is relevant to the SUT. For example, in Fig. 2, the attribute type of Item is used by the SortingController to move the Sorter in appropriate position before the items arrive. By using the profile, it is also possible to leave the decision of selecting exact values for properties of the environment components till the time of testing (where it is decided by the simulation configurations). This concept is modeled by assigning «NonDeterministic» to the properties of environment components. This stereotype has three properties: an upper bound, a lower bound, a valueConstraint, and a scope. The upper and lower bound specify the possible range of values that an integer property of an environment component can take during simulation. This is provided to ease the modeling of time events’ bounds. Alternatively, an OCL constraint can be provided as a valueConstraint that restricts the possible values that an environment property can take. This constraint can, for example, be used to restrict a string property to certain specific values. As shown in Fig. 1, the scope property can have three possible values: Class, State, or Dual. If the value is set to Class, the properties of the environment component instances are initialized with a value obtained from simulation configuration only once when the instances are created. If the value is set to State, the values are obtained whenever there is a state change in an instance. If the value of scope is set to Dual, then a value is obtained for this environment component’s property from the simulation configuration when an instance is created and the property is reassigned a value when there is a state change in the instance. For example, in Fig. 2, the property type of Item is a nondeterministic variable with the scope Class and its value is initialized based on a simulation configuration when an instance of Item is created. 5.2.4. Modeling the SUT Fig. 0. State machine for the Sorter component Simula Research Laboratory Technical Report (2011-04) Version 3.0 16 It is important to include the SUT in the environment domain model, so that its relationship with the other environment components can be specified. It is also useful to include the details of signal receptions by the SUT from other environment components. The SUT is stereotyped as «System». The stereotype was used initially by Gomaa [23] to refer the system in a context diagram. Since, our goal is software-in-the-loop testing, the SUT modeled in the domain model represents the SUT and its execution platform as a single component. 5.3. Behavior Modeling For each environment component in the domain model that has a behavior affecting the SUT, our methodology requires to create a state machine representing this behavior. The state machine captures such behavior at the level of abstraction that is visible to the SUT. The state machines are developed using UML 2.x state machine notation and concepts, MARTE real-time extensions, and our profile to assist in modeling the environmental aspects of RTES. The MARTE profile is used to model the features related to time and a form of non-determinism. As discussed earlier, during simulation, the instances of the environment components run in parallel to form the environment of the RTES. They can send signals to each other and to the SUT. We can also view the environment as having one state machine with orthogonal regions, one for each component. Fig. 4 Fig. 7 show the state machines of the four environment components of our motivating example. Note that the diagrams are developed in IBM RSA, which adds some additional symbols to the triggers and effects in the state machines. A change event is not shown with a when keyword as for example in the transition from On_Hold to No_Item in the ItemInside region of RVM state machine shown in Fig. 5. All the guards in the state machines have the corresponding environment components as the context for OCL constraints. Following, we discuss the details of the methodological guidelines for modeling behavior of the environment components. 5.3.1. Identifying Stateful Components Components whose states either affect the SUT or are affected by the SUT should be modeled with state machines.Overall, the environment should be modeled in a way that enables, after the Fig. 0. State machine of the RVM Component Simula Research Laboratory Technical Report (2011-04) Version 3.0 17 initialization and provision of simulation and environment configurations, the full simulation of the interactions with the SUT. All the environment components shown in Fig. 2 are stateful components of the sorting machine case study. For example, the Sorter component was modeled as stateful since it receives signals from the SortingController and reacts differently based on its current state. 5.3.2. States to be Included It is important to determine the right level of abstraction for a component state machine. If we want to precisely model the behavior of an environment component, this might lead to a large number of states. We are, however, only interested in state changes that have an impact on the SUT. A single state in an environment model state machine may correspond to a large number of concrete or physical states. For example, in the sorting machine, the states of Item that we modeled were all related to its movement through the sorting machine whereas its other possible states were not of interest as an environment component of the SortingController. A state in a UML state machine can be a simple state, a composite state (i.e., containing substates) or it can be a submachine state. UML state machines can also have multiple orthogonal regions. The concept of orthogonal regions is particularly useful in environment modeling as one environment component can in reality be composed of multiple sub-components. For example, RVM in our motivating example is composed of two sub-components: an item feeder that handles item insertion and a conveyer that is responsible for routing the items. From the perspective of the SUT (Sorting Controller) it is not important to distinguish these two components as it sees RVM as a single component. For the RVM, to completely simulate the behavior visible to the Sorting Controller, it must manage the movement of items on the conveyer in parallel to handling items in the feeder. From the RVM point of view, functionality must be provided for both of these components, conveyer and feeder. Therefore this information is modeled as two orthogonal regions of the RVM (named ItemInside and Routing) in the state machine shown in Fig. 5. In addition, according to our modeling methodology, failure behavior of a component that is independent of its nominal behavior can also be modeled as a separate orthogonal region. 5.3.3. Modeling Users in the Environment Fig. 0. State machine of Item environment component Simula Research Laboratory Technical Report (2011-04) Version 3.0 18 Generally, for software system modeling, users are only modeled as sources of inputs and data. The behaviors of users with respect to the system are mostly not considered. In the environment modeling methodology, it is useful to model the behavior of users in the environment to have a control over the inputs/outputs of the various components or the SUT. If a user participates in multiple roles, it is useful to model each role a user plays as a separate component. In the motivating example, we modeled the persons who enter the items for sorting as a User environment component (state machine shown in Fig. 7). In certain cases it can be interesting to model both the expected and unexpected behavior of users using the proposed methodology. Overall, the behavior of a user in an RTES environment is modeled using the same notations as any other environment component. 5.3.4. Modeling Events When using UML 2.x state machines for environment modeling, only three types of events are required to be modeled: signal events, time events, and change events. Call events are not required since the components in the environment represent active objects and communicate asynchronously. OCL is used to model guards on transitions and conditions in the change events. For example, Fig. 4 shows the state machine of the Sorter component. As discussed earlier, a Sorter can be at three different positions. This is represented by the three states, Left, Centre, and Right. Movement between these states is represented by the outgoing transitions from these states to the two movement related states: MovingLeftCentre and MovingCentreRight. For the Sorter to move from Left to Center it needs to transition first from Left to MovingLeftCentre, which is triggered on receiving a signal event POSITION_CENTRE() from the SUT (Sorting Controller). A transition from MovingLeftCentre to Centre state is triggered by the time event after “movingArmTimeLC, ms”, where movingArmTimeLC is the name of a non-deterministic property of Sorter and ms is the unit of time, milliseconds. This transition is only triggered if the guard on the transition, written in OCL (self.destination = “centre”), is true. An example of a change event can be seen in the state machine of the RVM component (Fig. 5) in the ItemInside region on a transition from On_Hold to No_Item. The transition has an effect ^this.user.rvm_sends_item() written in Java, which we chose as the action language as further discussed later. The MARTE TimedEvent concept is used to model all timeout transitions, so that it is possible for them to explicitly specify a clock (if needed). Each environment component may have its own clock Fig. 0. State machine of User Simula Research Laboratory Technical Report (2011-04) Version 3.0 19 or multiple components may share the same clock for absolute timing. The clocks are modeled using the MARTE’s concept of clocks. If no clock is specified (as in the case of motivating example), then by default the notion of time is considered to be according to the physical time. Specifying a threshold time for an action execution or for a component to remain in a state is possible using the MARTE TimedProcessing concept. This is also a useful concept and can be used, for example, to model the behavior of an environment component when the RTES expects a response from it within a time threshold. The proposed environment modeling profile allows the modeler to apply three stereotypes to transitions in the state machines: «NonLeaving», «TimeProbability», and «gaStep» (defined by MARTE profile). Following, we discuss the stereotype «NonLeaving», whereas the other two stereotypes are related to non-determinism and will be discussed later (Section 5.3.7). By default whenever a component transitions from one state to another (i.e., transitions that are not internal), its event queue is emptied. To control the effect of a transition on the event queue and timers of the context component, we defined the stereotype «NonLeaving». In other words, this stereotype is a way to give more control to the modeler over the internal handling of the queues and timers. The stereotype has a property called sideEffect, which can have three possible values: (1) NoSideEffects, to denote that the transition should have no side effects on the source state. A transition with this stereotype will result in no alteration of the event queue and the various timers in the environment component; (2) DoNotEmptyQueue, which will result in no alteration of the event queue, but will reset the timers; (3) DontResetTimers where the queue is emptied, but does not reset the timers. 5.3.5. Modeling Actions & Action Durations 1. import simula.embt.commons.*; 2. public class SortingMachineActionCode implements ExternalCode 3. { 4. private int port; 5. private String address; 6. private Connection con; 7. private IActiveObject sorter; 8. public SortingMachineActionCode(Object[] args) 9. { 10. port = (Integer) args[0]; 11. address = (String) args[1]; 12. } 13. public void openConnection() throws Exception 14. { 15. con = TCPConnection.openConnection(address, port, 1000); 16. String msg = con.readMessage(); 17. sorter.receiveSignal(msg, null); 18. } 19. public void triggerSensor() throws Exception 20. { con.sendMessage(Signals.TRIGGER_SENSOR); } 21. @Override 22. public void startExecution(IActiveObject ao) 23. { sorter= ao; } 24. @Override 25. public void stopExecution() throws Exception 26. { con.stopExecution(); } 27. } Fig. 0 Excerpt of ExternalActionCode for the Sorter component Simula Research Laboratory Technical Report (2011-04) Version 3.0 20 In our methodology, we chose Java as the action language for writing actions. The decision to choose Java as the action language at the model level is due to the current lack of tool support for the UML action language (ALF) [41] at the time of our tool development. Moreover, the testers of the SUT are expected to be more familiar with Java (consistent with our experience of applying the approach in two industrial contexts), rather than with a newly accepted standard language. In the environment models, actions can be written in two places. Simple actions can be written inside the models, e.g., in the RVM state machine (Fig. 5), the simple assignment action of the transition from ItemInside::No_Item to ItemInside::On_Hold (i.e., this.currentItem = item) is placed directly as an effect. Relatively complex actions and communication related details are written in a separate source file and are referred to as the external action code. Calls to methods of external action code classes are simply made by using the keyword action: action.. For example, in Fig. 5 the effect of the initial transition in the ItemInside region is in fact making a call to openConnection() in the action class corresponding to RVM (see line # 13 in Fig. 8). External action code is the code that is to be written manually by the tester in a separate source code file, to communicate with the SUT and compute complex effects (e.g., action code computing continuous physical effects could also be generated by modeling and simulation tools, such as Modelica [10]). An example for the type of external action code is signals transmitted to the SUT over a UDP/TCP communication layer. An excerpt of the external action code for the Sorter component is shown in Fig. 8 (line # 13 and line # 19). The action code for the two messages sent to the action object in the state machine of Sorter (Fig. 4) openConnection() and triggerSensor() – can also be seen in the excerpt shown. Class TCPConnection is part of the communication library that we used. The method triggerSensor() simply forwards the signal to the SUT over the TCP connection. The decision to keep such action code separate was made to avoid cluttering the models with unnecessary details and to allow developers to write this code in a familiar programming tool. It was also important to keep the communication related information separate to avoid changing the models in case of changes in the communication mechanism. For example, if we want to change the communication from TCP to UDP, the only change will be in the external action code classes. Given a communication layer, even if the simulator is generated in Java, there is no particular restriction on the programming language in which the SUT is implemented. To provide a mapping between the environment components and corresponding classes containing the external action code, an External Code Mapping file is provided by the modeler. Fig. 9 shows an excerpt of this file for the sorting machine case study. The file contains the mapping details of external action code and environment components for the two components (Sorter and RVM) that are tomra.embt.env.Sorter tomra.embt.action.SortingMachineActionCode tomra.embt.env.RVM tomra.embt.action.SortingMachineActionCode Fig. 0 Excerpt of External Code Mapping File for the Sorting Machine case study Simula Research Laboratory Technical Report (2011-04) Version 3.0 21 communicating with the SUT. For the other two classes, no action class was required. This information could have also been placed in the models, but we decided to keep things related to external action code separate from the models, so that it can be changed without affecting the models. Specifying a time threshold for an action execution or for a component to remain in a state is possible using the MARTE TimedProcessing concept. This is also a useful concept and can be used, for example, to model the behavior of an environment component when the RTES expects a response from it within a time threshold. Though in our case studies we did not face a situation where we needed to model action durations, the methodology supports this feature. 5.3.6. Modeling Error & Failure states Two of the important features that are modeled in the state machines of environment components are the Error and Failure states. Failure states represent possible failures in the environment of SUT, e.g., hardware failure in the components. These states are required to test the robustness of the SUT when confronted to failures in the environment components. The failure states are modeled with the «Failure» stereotype. Failures that are independent of any specific aspect of an environment component’s behavior (e.g., a hardware failure that can occur in any component state) are typically modeled using separate parallel regions within the state machine of the environment component. Error states are the states of the environment that can only be reached due to faulty behavior of the SUT. These states are conditions that should never happen in the environment, otherwise indicating that the SUT is faulty. For example, a Sorter should never receive an item while it is moving and when there are no simulated failures in the hardware of the environment components, all items should always be delivered to the correct destinations based on their types. It is the responsibility of the Sorting Controller (SUT) to make the Sorter reach the appropriate position before an item reaches it. Otherwise, this would mean that there is a fault in the implementation of the Sorting Controller. This behavior of Sorter is modeled in the state machine shown in Fig. 4 as an error state, which is labeled with «Error». Error states are key oracle information that is used during the test execution of the SUT. By modeling erroneous situations as states, the methodology allows modeling of erroneous situations due to violation of temporal constraint (modeled as time transitions leading to error states), due to illegal change in the state of the environment (modeled as transitions leading to error states triggered by change events), and due to erroneous signal receptions (modeled as a transitions leading to error states triggered by signal events). 5.3.7. Modeling Non-Determinism Non-determinism is a particularly important concept for environment modeling and is one of the fundamental differences between models for system modeling and models for environment modeling. Following we discuss different types of non-determinism that we have modeled for our case studies. For a number of RTES environment components, specifying the exact values for timeout transitions is not always possible. To model their behavior in a realistic way, it is often more Simula Research Laboratory Technical Report (2011-04) Version 3.0 22 appropriate to specify a range of values for a possible timeout, rather than an exact value. Moreover, the behavior of humans interacting with the RTES is by definition non-deterministic. For modeling these behaviors, the modeler can add an attribute in the environment component and label it with the stereotype «NonDeterministic». The stereotype allows the modeler to provide an upper and lower bound values by directly specifying them in the properties or by using an OCL constraint to restrict the possible set of values of the attribute. This attribute can then be used as a parameter of a time event. In the state machine of the User (Fig. 7), the transition between the state Idle and RVM_Busy is modeling the behavior that this transition is non-deterministic and that the user can insert the next item with a delay ranging from 1 to 10,000 milliseconds. The information constraining the values is provided in the domain model by applying the stereotype «NonDeterministic» on the insertionTime attribute (see Fig. 2). The attribute is then used as a parameter to the time event on the transition. The actual value (between the range specified) to be used during simulation is obtained from the Simulation Configuration. Since the scope property of the stereotype is set to State, the value for this attribute will be obtained from the Simulation Configuration every time the User enters Idle state, i.e., every time a new item is inserted. There can be situations in which the modeler wants to restrict that a non-deterministic value is either only obtained once for each instance (i.e., assigned at the time of instantiation) or is obtained at the time of instance creation and every state change. These restrictions can be modeled by setting the property scope of the stereotype «NonDeterministic» to Class or Dual respectively. For example, The attribute type for an Item (see Fig. 2) on the basis of which the Item is sorted is modeled as «NonDeterministic» with scope set to Class and values constrained between 0 and 2 representing different types of items. This means that for each instance of Item, the attribute type is given a value by a simulation configuration when an instance of Item is created. Another important form of non-determinism is to assign probabilities to the transitions of state machines. In an RTES environment, we sometimes only know the probability of a component to go into a particular state over time and we are not sure about the exact occurrence of such conditions. For example, we can say that the probability of a car engine to overheat after running continuously for 10 hours is 0.05, but we cannot be certain about the exact instance in time when this situation will happen. We can model this in the engine state machine with a transition going from Normal Temperature state to Overheated state, during an interval of 10 hours, with probability of 0.05. For modeling these scenarios, we can assign a probability on the transitions using the property prob of the MARTE GaStep concept. Whenever a timeout transition has the gaStep stereotype applied with a non-zero value of prob, the combination will be comprehended as the probability of taking the transition over time of the test case execution. In the sorting machine case study, a Sorter can get stuck in a position (e.g., because of a bottle blocking it or the arm jamming) for example with a probability 0.02 in a minute if it is not moving and a higher probability when it is moving. The Simula Research Laboratory Technical Report (2011-04) Version 3.0 23 sending of non-deterministic signals can also be modeled using this type of transitions, by placing them in the actions of such transitions. If the goal is validation, for example based on reliability estimation, then these probability values can be used as a sort of operational profile of the SUT [42]. On the other hand, if the goal of testing is the verification of the SUT, then the actual values of these probabilities are not important (Test Framework decides if an event happens, as long as its probability is not zero). For example, if the goal was validation, the above discussed scenario of a Sorter getting stuck could have been modeled with the gaStep stereotype to provide an exact value, range, or a probability distribution of occurrence of this failure. For verification purposes, typically we only require modeling of such situations without specifying an exact probability value or distribution and leave the decision of exact value to the Test Framework. The stereotype «TimeProbability» on a transition is used to model such a non-deterministic trigger, whose occurrence is decided by the Test Framework and obtained from the simulation configuration. Such a transition is very useful to represent failures in environment components. For example, in the Sorting machine case study, an item can fall of the belt and be lost at any time while it is moving inside the machine. This is modeled as a transition from On_Belt state to a failure state named Lost in the Item state machine shown in Fig. 6. Another type of probability that we modeled in our case studies is for the situations where one event can lead to multiple possible scenarios, but all of them are mutually exclusive. For example, we might want to represent the fact that during the communication with the SUT there is a chance that signals are received with or without distortion. For modeling such scenarios in UML state machines, we can use choice nodes. We provided a stereotype «NDChoice» that can be applied on choice nodes, where each outgoing transition has the same probability. The decision of taking one of the outgoing transitions from such a choice node is made at the time of execution by the Test Framework. If the modeler wants to provide precise probabilities for such scenarios, she can assign the MARTE gaStep stereotype to each of the multiple possible outgoing transitions. The example of communication with the SUT can be modeled by having two transitions going out of the environment component state on receiving of a signal, one labeled with a probability that the signal was corrupted and the other with the probability that the signal was fine. As mentioned earlier, modeling the distribution of event arrivals and timeout transitions can be useful for validation purposes, but is out of the scope of this paper, since our goal is verification of the SUT. 6. Simulator Generation The environment models, comprising a domain model (UML class diagram) and behavioral models (UML state machines), are converted into a Java-based simulator using model to text transformations. The transformations are based on an extension of the state pattern [7], which is a well-known way of implementing state machines. The transformations proposed here are defined to Simula Research Laboratory Technical Report (2011-04) Version 3.0 24 address the specific requirements for environment simulation and RTES system testing. In this section, we first provide an overview of the overall simulation framework. Then we discuss our extended state pattern followed by a discussion on detailed transformation rules for domain and behavior models to simulator code, thus providing a more thorough description of the pattern. 6.1. Simulation Framework Fig. 10 shows the architecture of the Simulation Framework. Components marked with the stereotype «artifact» represent the artifacts that are provided by the software testers to use the framework. The two inputs include the Environment Models that are developed according to the methodology discussed in Section 5 and the External Code Mapping file that provides a mapping between the external action code and environment models (as discussed in Section 5.3.5) The package named Simulator Generator contains the core components required for simulator generation. The sub-package GeneratorDrivers contains driver classes provided with the framework that are responsible for configuring and running the model transformations. The Model Transformations package contains the transformations we wrote in MOFscript [43] to translate the environment models to Java classes representing the environment simulator. Class EnvironmentConfigurationGenerator is responsible for generating an environment configuration representing one possible setting of the environment. The class OCLToJavaTranslator is used by the MOFScript transformations to translate the OCL expressions in the model representing guards and change events to their Java equivalent. More details on how the simulator generator handles change events are provided later in Section 6.4.1. The components inside the Simulator Generator package generate a set of classes in Java corresponding to the environment models given as input. This is represented as a Simulator package in Fig. 10. The generated simulator is statically linked to classes from two packages: the Simulator Helper Library (SHL) and the Non-Deterministic Engine which we discuss below. The Simulator Helper Library is developed to support a number of features required by the generated simulator. The library is independent of the case studies and hence is developed as a separate library. The library contains generic features required by active objects (message queue, event handling mechanism, etc.), time related functionalities (including features for handling clocks Fig. 0 Architecture Diagram of Simulation Framework Simula Research Laboratory Technical Report (2011-04) Version 3.0 25 and timed events), collection classes (providing facility for sending broadcast signals to all elements in the collection), and support for implementing the defined extension of the state pattern, as discussed further in Section 6. The core package of the library is shown in Fig. 11. The class ActiveObject represents the UML concept of an active object. The class provides an event queue for the active objects in the form of a java.util.PriorityBlockingQueue. The queue is a blocking queue and enables setting the priority of elements in the queue. The queue holds instances of class EventInvocation. An EventInvocation instance represents an event that is ready to be executed and is placed in the event queue of an ActiveObject. EventInvocation has an attribute methodToInvoke that is of type Method from the Java reflections package and contains the method of the ActiveObject to be invoked along with its parameter types, an attribute parameterValues containing the values for the parameters of the method to be invoked, an attribute triggerType representing the type of the trigger (signal event, change event, or time event), and an attribute isTimeProbable that indicates whether the method to be invoked represents a time probability trigger. The class ActiveObject is an abstract class containing the generic behavior of an active object. The behavior provided by the class is used both for the environment components (extending the further generalized class SimulatedObject) and implementation of parallel regions (since each region has its own thread of execution and an event queue). The interface IState is implemented by all the classes representing UML states. The class SimulatorList is used to implement relationships having a multiplicity greater than 1. The class provides facility to broadcast events to all the elements it contains at any given time. For example SimulatorList is used to implement the relationship (itemCollection, Fig. 2) between Item and User for the Sorting Machine case study, so that signals to items can be easily broadcasted. The Non-Deterministic Engine is responsible to provide a link between the simulator and various simulation configurations produced by the Test Framework. The Non-Deterministic Engine is called by the simulator each time a non-deterministic occurrence needs to be produced, which in turn queries Fig. 0. Important classes in the SimulatorHelperLibrary Simula Research Laboratory Technical Report (2011-04) Version 3.0 26 the current simulation configuration and returns the value generated by the Test Framework corresponding to the non-deterministic occurrence. This is handled by assigning a unique id to each non-deterministic occurrence during the entire simulation, based on the formula: = * + , where NDID is the non-deterministic occurrence id to be calculated, INSTANCE_ID is the unique id assigned to instances of environment components, MAX_ND_COUNT is the maximum number of non-deterministic occurrences that any component in the domain model can have, and LOCAL_ND_ID is the unique id for the occurrence for each environment component. For example, in the Item component state machine (Fig. 6), the transition from On_Belt to Lost is stereotyped as «TimeProbability», which is a non-deterministic event. The actual value for the time when this transition is to be taken is obtained by the simulator through the non-deterministic engine. As discussed later in Section 6.4.3, non-determinism can be of multiple types. An excerpt of generated code in Fig. 12 shows a call to the Non-deterministic Engine in order to initialize the value of the attribute type of the Item environment component. The statement instanceId * 3 + 0 will result in a unique id for this non-deterministic occurrence during